Augmented view of advertisements

ABSTRACT

A head-mounted display (HMD) provides an augmented view of advertisements to an HMD wearer. In some embodiments, when an advertisement is within an HMD wearer&#39;s field of view, the HMD may augment the HMD wearer&#39;s view of the advertisement to provide additional information and/or to personalize the advertisement to the HMD wearer. In other embodiments, when an advertisement is within an HMD wearer&#39;s field of view, the HMD may augment the HMD wearer&#39;s view of the advertisement to remove the advertisement from the HMD wearer&#39;s view or to replace the content of the advertisement with non-advertising content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/041,775, filed Mar. 7, 2011, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND

Advertising has become a pervasive part of our society, which may resultin people sometimes feeling inundated with advertisements. Additionally,people may often feel that they have no interest in the product orservice advertised. A number of factors may contribute to this. Forinstance, advertisements typically provide a static advertising messagethat is often not relevant to many people or otherwise engages them.Although some electronic billboards and other display advertisements maychange the advertising message being displayed, they are still limitedin what advertisements are shown. As a result, people may simply tuneout the advertisements at best or actually become annoyed with theadvertisements at worst, and in turn, advertisers may be less likely tosee conversions from their advertisements. Further, when people actuallywould like advertising information regarding a product or service, theymay not be able to access that information. For instance, people may seeproducts in their everyday lives in which they are interested but maynot be able to determine who sells the products or may not be able toget information regarding the products when they see them. Thesesituations present excellent opportunities for advertising thatunfortunately are lost.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Embodiments of the present invention relate to employing head-mounteddisplays (HMDs) to target advertising and other information to HMDwearers. In some embodiments, an advertisement in the real world may beidentified within an HMD wearer's field of view. The HMD wearer's viewof the advertisement may be augmented by the HMD to provide additionalinformation and/or to personalize the advertising message to the HMDwearer. In further embodiments, the HMD wearer may be provided a servicethat removes or replaces advertisements in the real world from the HMDwearer's view. This may be, for instance, a fee based or othermembership-based service provided to the HMD wearer. In suchembodiments, when an advertisement is identified in the HMD wearer'sfield of view, an augmented view may be provided by the HMD to removethe advertisement from the HMD wearer's view or to replace the contentof the advertisement with non-advertising content.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor use in implementing embodiments of the present invention;

FIG. 2 is a flow diagram showing a method for providing advertisementsvia an HMD based on an object viewed by an HMD wearer's focal point inaccordance with an embodiment of the present invention;

FIG. 3 is a flow diagram showing a method for employing an HMD todisplay information regarding a person being viewed by the HMD wearer inaccordance with an embodiment of the present invention;

FIG. 4 is a flow diagram showing a method for providing information toan HMD wearer based on other HMD wearers viewing a particular object orperson in accordance with an embodiment of the present invention;

FIG. 5 is a flow diagram showing a method for employing an HMD toaugment an HMD wearer's view of an advertisement to personalize theadvertisement in accordance with an embodiment of the present invention;and

FIG. 6 is a flow diagram showing a method for employing an HMD toaugment an HMD wearer's view of an advertisement to remove or replacethe advertisement in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Embodiments of the present invention are directed to providing targetedadvertising and other information via a head-mounted display (HMD). Inaccordance with some embodiments, an advertising system for an HMD maytarget information based on parameters indicative of the HMD wearer'scurrent interest and other factors from the HMD wearer's currentenvironment. The system may attempt to determine what the HMD wearer maybe currently thinking about and interested in to target advertisements.The HMD wearer's gaze may be tracked to determine the HMD wearer'sfocus. Surrounding data may be faded out and focus placed on the HMDwearer's focal point to collect data points for advertisement selection.The HMD wearer's length of gaze may also be tracked. The longer an HMDwearer gazes at a particular subject (person, place, thing), thestronger the data point becomes. More and more information regarding thesubject is brought in for advertisement selection as the HMD wearercontinues to look at the subject, and more and more information isprovided to the HMD wearer. In some embodiments, the HMD wearer may belooking at a particular person. The system may identify the person andbring in information about the person, which in some embodiments may beused in advertisement selection. In further embodiments, the system mayuse information about the focus of multiple other HMD wearers foradvertisement selection. In particular, if multiple people in the HMDwearer's vicinity are looking at a particular subject, it's likely thatthe subject is interesting or important for some reason.

In some embodiments, an HMD wearer's view of an existing advertisement(e.g., a billboard) may be augmented by the HMD. In some instance, theadvertisement may be personalized using the HMD to augment the HMDwearer's view. The advertisement may have a generic message for peoplewho are not wearing an HMD. For people wearing an HMD, the view of theadvertisement may be changed to personalize the advertisement. This mayinclude augmenting portions of the advertisement or re-skinning theentire advertisement with a personalized message. One way anadvertisement may be personalized would be placing an image of the user,the user's family, and/or the user's friends in the context of theadvertisement.

In still further embodiments of the present invention, an HMD wearer maybe provided the option to have advertisements removed from the HMDwearer's view or replaced with other content. In some embodiments, theadvertisements may be simply removed. In other embodiments, theadvertisements may be replaced with the HMD wearer's own content orother non-advertising content. For instance, an advertisement on abillboard could be replaced with a picture of the HMD wearer's family.

Accordingly, in one aspect, an embodiment of the present invention isdirected to a method to remove an advertisement from an HMD wearer'sview of the real world. The method includes identifying theadvertisement within the HMD wearer's field of view. The method alsoincludes causing an HMD device to display a computer-generated virtualimage to remove the advertisement from the HMD wearer's view of the realworld.

In another embodiment, an aspect of the invention is directed to one ormore computer storage media storing computer usable instructions that,when used by one or more computing devices, cause the one or morecomputing devices to perform operations. The operations includeidentifying an advertisement within an HMD wearer's field of view of thereal world. The operations also include causing an HMD device to displaya computer-generated virtual image to remove the advertisement from theHMD wearer's view of the real world.

A further embodiment of the invention is directed to a system thatincludes one or more processors; and one or more computer storage mediastoring computer usable instructions that, when used by the one or moreprocessors, cause the one or more processors to: identify anadvertisement within an HMD wearer's field of view of the real world;and cause an HMD device to display a computer-generated virtual image toremove the advertisement from the HMD wearer's view of the real world.

Having briefly described an overview of embodiments of the presentinvention, an exemplary operating environment in which embodiments ofthe present invention may be implemented is described below in order toprovide a general context for various aspects of the present invention.Referring initially to FIG. 1 in particular, an exemplary operatingenvironment for implementing embodiments of the present invention isshown and designated generally as computing device 100. Computing device100 is but one example of a suitable computing environment and is notintended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing device 100be interpreted as having any dependency or requirement relating to anyone or combination of components illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Theinvention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 1, computing device 100 includes a bus 110 thatdirectly or indirectly couples the following devices: memory 112, one ormore processors 114, one or more presentation components 116,input/output (I/O) ports 118, input/output components 120, and anillustrative power supply 122. Bus 110 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 1 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Theinventors recognize that such is the nature of the art, and reiteratethat the diagram of FIG. 1 is merely illustrative of an exemplarycomputing device that can be used in connection with one or moreembodiments of the present invention. Distinction is not made betweensuch categories as “workstation,” “server,” “laptop,” “hand-helddevice,” etc., as all are contemplated within the scope of FIG. 1 andreference to “computing device.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing device 100. Communication mediatypically embodies computer-readable instructions, data structures,program modules or other data in a modulated data signal such as acarrier wave or other transport mechanism and includes any informationdelivery media. The term “modulated data signal” means a signal that hasone or more of its characteristics set or changed in such a manner as toencode information in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer-readable media.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 100includes one or more processors that read data from various entitiessuch as memory 112 or I/O components 120. Presentation component(s) 116present data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 118 allow computing device 100 to be logically coupled toother devices including I/O components 120, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc.

As previously mentioned, embodiments of the present invention may beimplemented, at least in part, by an HMD. The HMD may include componentssuch as those described above with reference to FIG. 1 for the generalcomputing device 100, including, for instance, memory, processors,computer-readable media, input/output components, and a power supply.The HMD may be provided in any of a number of different form factors,including, for instance, glasses, goggles, or a helmet.

The HMD may generally provide an augmented view of the real world to awearer by augmenting the wearer's view of the real world withcomputer-generated virtual images. To provide the wearer with anaugmented view, the HMD may include display technology that displayscomputer-generated virtual images while still allowing the wearer toview the real world. In some embodiments, this may include heads-updisplay (HUD) technology, which may employ, for instance, any type ofprojection or microdisplay technology to provide the virtual images.Other technologies may also be employed, such as retinal displays inwhich images are projected directly on the wearer's retina while thewearer is viewing the real world.

In some embodiments, the HMD may align computer-generated virtual imageswith the HMD wearer's view of the real world as the wearer's view of thereal world changes. To assist in locating virtual images in the wearer'sview of the real world, the HMD may include devices that may monitorlocation and/or provide head, eye, and/or gaze tracking. These devicesmay include, without limitation, GPS or other location trackingcomponents, cameras or other optical sensors, gyroscopes, andaccelerometers, to name a few.

The HMD may also include wireless communication capabilities to allowthe HMD to communicate with companion devices (e.g., smartphones),server devices, or other network components. For instance, someembodiments of the present invention may be implemented by having acloud-based service assist the HMD in selecting virtual images toprovide an augmented view of the real world. In some embodiments, theHMD may be configured to communicate directly with server devices; whilein other embodiments, the HMD may communicate with server devices via acompanion device local to the HMD wearer (e.g., a smartphone or otherdevice).

Advertising Based on HMD Wearer's Focal Point

Some embodiments of the present invention are directed to providingadvertisements based on the person's current interests and thoughts bytracking the person's gaze using an HMD to determine the person's focus.Instead of using data regarding objects within the HMD wearer's entirefield of view, an object within the HMD wearer's actual focal point isdetermined and advertising information is provided based on that object.As is known in the art, a field of view includes the entire extent ofwhat a person sees at a given moment. This is in contrast to a person'sfocal point, which is a small area within the field of view that is infocus. For instance, suppose an HMD wearer is located on a street with anumber of restaurants and shops that are all within the HMD wearer'sfield of view. However, the HMD wearer may be looking at the sign of aparticular shop such that the HMD wearer's focal point is on the sign.Embodiments of the present invention may determine that the HMD wearer'sfocal point is on the sign, obtain advertising information based onidentifying the sign within the HMD wearer's focal point, and presentthe advertising information via the HMD.

The HMD wearer's length of gaze may also be tracked and employed foradvertisement selection and display. The amount of time an HMD wearer'sfocal point remains on a particular object is indicative of differentlevels of interest in the object being viewed. The longer a person gazesat a particular object, the more interest the HMD wearer likely has inthat object and the stronger the data point becomes for advertisementdelivery.

In some embodiments, a minimum threshold period of time may need to besatisfied before any advertising information is provided based on agiven object. For instance, an HMD wearer may pan over a particular areawith the HMD wearer's focal point quickly moving from object to object.In such a situation, it may be undesirable to display advertisinginformation for each of the objects as this may present clutter.Additionally, since the HMD wearer is not focusing on any particularobject for more than a quick glance, the objects are not likely ofparticular interest to the HMD wearer such that advertising informationbased on the objects would not be particularly targeted. However, oncean HMD wearer's focal point remains on a particular object for theminimum threshold of time, advertising information may be selected basedon that object and presented to the HMD wearer.

Different amounts of advertising information may also be presented basedon the HMD wearer's length of gaze on a given object. For instance, insome embodiments, a number of threshold periods of time may beestablished. As each threshold is successively reached, additionaladvertising information is presented to the HMD wearer. In furtherembodiments, instead of establishing various thresholds at whichadditional information is provided, more advertising information iscontinuously added to the HMD wearer's view as the HMD wearer's focalpoint remains on a given object. In some instances, an HMD wearer mayglance away from an object and then return to the object. The system maybe able to recognize that the HMD wearer's interest has primarilyremained on the object, and the system may ignore the glance away andtreat the situation as if the HMD wearer continued to view the object.

In some embodiments, a number of objects that are viewed within the HMDwearer's focal point over a period of time may be used as aggregateddata for advertising selection. For instance, an HMD wearer may look atthe sign of one restaurant and then view the sign of another restaurant.This information may indicate that HMD wearer is looking for a place toeat. Based on this aggregate data, the system may select advertisinginformation regarding other restaurants in the HMD wearer's vicinity anddisplay the information to the HMD wearer.

Additional data signals may be used for advertisement selection inconjunction with an object within an HMD wearer's focal point. By way ofexample only and not limitation, additional data signals that may beused may include the HMD wearer's biometrics or a user profile for theHMD wearer. For instance, in some embodiments, a user profile may bemaintained for the HMD wearer and used in conjunction with an objectbeing viewed to select advertising information to display to the HMDwearer. The user profile may include a variety of information regardingthe HMD wearer, such as the HMD wearer's demographics, preferences, andinterests. For instance, suppose that an HMD wearer is looking at a signfor a restaurant and the HMD wearer's user profile includes a list ofrestaurants that the HMD wearer prefers. Based on these data points, thesystem may determine that a preferred restaurant from the user profileis near the HMD wearer. As such, the system may deliver advertisinginformation for that preferred restaurant to the HMD wearer, includingthe restaurant's location.

In various embodiments of the present invention, the object being viewedmay be either a non-advertisement object or an advertisement. Anadvertisement may include anything which has the primary purpose ofproviding an advertising message. For instance, advertisements couldinclude billboards, press advertisements (e.g., in magazines,newspapers, etc.), and in-store advertisements, to name a few. Incontrast, a non-advertisement object could be anything else that doesnot have advertising as its primary purpose. For instance, the objectcould be something that could be purchased, such as clothing, electronicdevices, etc. Additionally, the object could be a bar code or othermachine-readable code. As another example, the object could be text readby the HMD wearer. Other objects, such as buildings and monuments, couldalso be non-advertisement objects that when viewed by an HMD wearer'sfocal point may trigger the display of advertising information via theuser's HMD.

In instances in which the object is an advertisement, the system mayretrieve and display additional information regarding the advertisedmessage. For instance, if the advertisement is a billboard advertising abrand of shoes, the system could retrieve pricing information for theshoes at various retailers. In addition to retrieving and displayingadditional information, the system could also track user interactionwith the advertisement. The information could include, for instance, howlong the HMD wearer looked at the advertisement, where the HMD wearer'sfocal point dwelled on the advertisement, and whether the HMD wearerinteracted with any of the additional information provided via the HMD.This advertisement information could be collected by the HMD andcommunicated to any advertising system for analysis and use of the datafor advertising purposes. For instance, the information could be used toanalyze the effectiveness of an advertisement and the information couldbe provided to the advertiser.

Turning to FIG. 2, a flow diagram is provided that illustrates a method200 for providing advertisements via an HMD based on an object viewed byan HMD wearer's focal point in accordance with an embodiment of thepresent invention. Initially, as shown at block 202, the HMD wearer'sfocal point is determined. The HMD wearer's focal point may bedetermined using any of a variety of techniques for eye and gazetracking. For instance, in one embodiment, the HMD may be equipped withoutward-facing and inward-facing cameras. The outward-facing camera maybe configured to capture the outside world around the HMD wearer. Theinward-facing camera may be configured to track the HMD wearer's eyes todetermine the line of gaze of the HMD wearer. Using images/video fromthese cameras, the HMD may determine where in the outside world the HMDwearer's focal point is located. As another example, in anotherembodiment, the HMD wearer's focal point may be determined by capturingan image on the HMD wearer's retina. It should be understood that theseare only examples and other techniques for determining the HMD wearer'sfocal point may be employed within embodiments of the present invention.

An object being viewed by the HMD wearer's focal point is determined atblock 204. Additionally, the length of time the object is viewed by theHMD wearer's focal point is determined, as shown at block 206. Adetermination is made at block 208 regarding whether the amount of timethe object is viewed by the HMD wearer satisfies a minimum threshold. Ifso, object recognition is performed at block 210 to identify the object.For instance, if the HMD wearer is viewing a car of a particular makeand model, the object recognition will determine that the object is acar of that make and model. It should be understood that the objectrecognition may be performed at block 204 when the object is initiallydetermined. Additionally, it should be understood that any of a varietyof known object recognition techniques may be used to identify theobject. Further, object recognition may be performed by the HMD, aserver device, or a combination. For instance, in one embodiment the HMDmay transmit image data to a cloud service that performs objectrecognition.

After the object being viewed by the HMD wearer's focal point isidentified, advertising information is retrieved based on the identifiedobject, as shown at block 212. In some embodiments, the advertisementselection may be performed by the HMD itself if the HMD has thecapabilities to locally cache advertisements. In other embodiments, theHMD may communicate with another device local to the HMD wearer (e.g., asmartphone or a dedicated ancillary device) to select advertisements. Instill other embodiments, the HMD may communicate with an advertisementdelivery system (e.g., a cloud-based service) either directly or viaanother device local to the HMD wearer to retrieve advertisinginformation. For instance, the HMD may provide information identifyingthe object being viewed (or just images of the object), and theadvertisement delivery system may select advertising information andprovide the advertising information to the HMD. In further embodiments,the advertisement information may be selected based on other parameters,such as the HMD wearer's biometrics, user profile information for theHMD wearer, and/or past advertisements delivered to the HMD wearer. Theadvertising information selected may also be based on the length of timethe HMD wearer views the object.

The advertising information is presented via the HMD, as shown at block214. Typically, the advertising information may be displayed via theHMD. For instance, the HMD may include HUD technology or retinal displaytechnology to allow the HMD wearer to view the information. Inembodiments in which the HMD includes speakers, the advertisinginformation may include audio that is played by the speakers. Any andall such combinations are contemplated to be within the scope ofembodiments of the present invention.

As discussed previously, in some embodiments, if the HMD wearercontinues to view a particular object, additional advertisinginformation may be presented to the HMD wearer. Accordingly, the method200 may continue by determining if the HMD wearer's focal point remainson the object, and if so, additional advertising information isretrieved and presented via the HMD.

It should be understood that the method 200 may be performed by an HMDalone or in combination with one or more companion devices local to theHMD wearer (e.g., a smartphone or a dedicated ancillary device) and/ornetwork services (e.g., a cloud-based service). Any portion of themethod 200 described above may be performed by any combination of thesedevices and services.

Although the method 200 discussed retrieving advertising informationonly after the HMD wearer views the object for a threshold period oftime, in some embodiments, the HMD may retrieve advertising informationin the background for objects in the HMD wearer's field of view orotherwise known to be in the vicinity of the HMD wearer. This mayimprove system latency. In particular, the advertising information isreadily available and can be displayed when the HMD wearer's focal pointremains on an object for the threshold period of time.

HMD Information Based on Person being Viewed

While the embodiments discussed above provide advertising informationbased on objects being viewed, further embodiments select and displayinformation via an HMD based on a person being viewed by an HMD wearer.Generally, when an HMD wearer views a person, information is retrievedand presented to the user via the HMD. In some embodiments, theinformation may be statically displayed over to the side separate fromthe person being viewed, such that the information remains at thelocation displayed regardless of the HMD wearer movements. In otherembodiments, the information may be displayed in conjunction with theperson being viewed, such that the information may moves with thatperson. In still further embodiments, information may be audiblypresented via speakers on the HMD.

The information that may be retrieved and presented based on the personbeing viewed may include advertising information and/or personalinformation regarding the person being viewed. With respect to theformer, any of a variety of different advertising information may bepresented based on the person being viewed. By way of example only andnot limitation, advertising information regarding what clothing andaccessories the person is wearing may be provided. As another example,advertising information regarding where the person had his/her hair ornails done could be provided.

In some embodiments, a person may specify information regarding productshe/she owns or services the person receives to facilitate the deliveryof advertising information when the person is viewed by an HMD wearer.This could be viewed as a “word of mouth” approach of spreadinginformation regarding products and services in that an HMD wearer mayfind out about products or services when viewing a person. In someembodiments, advertisers may provide incentives to a person if theperson includes the advertisers' products or services in the list ofproducts and services the person would like to share with others. Forinstance, an advertiser could provide rebates or other incentives for aperson to simply include the advertiser's product or services in theperson's list of products and services that may be viewed by HMD wearerswhen viewing the person. In other instances, the amount of rebates orother incentives may be based on the number of HMD wearers who arepresented with information regarding the advertiser's product or servicewhen viewing the person. In some cases, the amount of rebates or otherincentives may be based on the number of HMD wearers who take furtheraction after viewing the advertising information, such as purchasing theadvertiser's product or service. Any and all such variations arecontemplated to be within the scope of embodiments of the presentinvention.

As noted above, personal information regarding a person being viewedcould also be displayed via the HMD. The personal information maygenerally include any personal information regarding the person that maybe of interest to others. In some embodiments, the information providedmay be based on the context of the interaction between the HMD wearerand the person being viewed. For instance, different types ofinformation may be appropriate for social interactions while other typesof information may be appropriate for business interactions.

In some embodiments, a person is given the ability to also controlaccess rights of HMD viewers to information associated with that person.This may include controlling which HMD wearers are able to accessinformation for that person. Additionally, the person may be able tocontrol what information different HMD wearers are able to access andview for that person. The access rights could be based on, for instance,individuals specifically identified by the person, individuals includedin the person's contact list, and/or individuals who have somerelationship with the person, such as being affiliated (e.g., “friends”)in a social network.

While people may be able to control what information HMD wearers seewhen viewing them, HMD wearers may be able to limit what information isdisplayed. For instance, a large amount of information regarding aperson may be available to an HMD wearer. However, the HMD wearer mayonly wish to see certain pieces of information. Accordingly, the HMDwearer may establish preferences regarding the type of information theHMD wearer would like to see. These preferences could be used to filterthe information that is presented to the HMD wearer when viewing aperson.

The HMD wearer's length of gaze on a person may also be tracked andemployed for controlling the display of information. The amount of timean HMD wearer views a particular person is indicative of differentlevels of interest in the person being viewed. The longer a person gazesat a particular person, the more interest the HMD wearer likely has inthat person.

In some embodiments, a minimum threshold period of time may need to besatisfied before any information is provided based on a given person.For instance, an HMD wearer may pan over a particular area with the HMDwearer quickly viewing one person after another. In such a situation, itmay be undesirable to display information for each person as this maypresent clutter. Additionally, since the HMD wearer is not focusing onany particular person for more than a quick glance, the people are notlikely of particular interest to the HMD wearer. However, once an HMDwearer views a particular person for the minimum threshold of time,information may be selected based on that person and presented to theHMD wearer.

Different amounts of information may also be presented based on the HMDwearer's length of gaze on a given person. For instance, in someembodiments, a number of threshold periods of time may be established.As each threshold is successively reached, additional information ispresented to the HMD wearer. In further embodiments, instead ofestablishing various thresholds at which additional information isprovided, more information is continuously added to the HMD wearer'sview as the HMD wearer continues to view the person. In some instances,an HMD wearer may glance away from a person and then return to theperson. The system may be able to recognize that the HMD wearer'sinterest has primarily remained on the person, and the system may ignorethe glance away and treat the situation as if the HMD wearer continuedto view the person.

With reference now to FIG. 3, a flow diagram is provided thatillustrates a method 300 for employing an HMD to display informationregarding a person being viewed by the HMD wearer in accordance with anembodiment of the present invention. Initially, a person being viewed bythe HMD wearer is identified, as shown at block 302. A person may beidentified using a variety of different techniques. For instance, insome embodiments, facial recognition may be employed to identify theperson being viewed. In such embodiments, the HMD may include anoutward-facing camera to capture an image/video of the person. The HMDmay perform the facial recognition or may communicate capturedinformation to a server to perform the facial recognition. In otherembodiments, the person being viewed may have an HMD or other device(e.g., a smartphone) that broadcasts information identifying the person.The HMD may detect the broadcasted information and use the informationto detect the person.

In some embodiments of the present invention, information may only beprovided for a person who is viewed by the focal point of the HMD wearerfor a threshold period of time. This may reduce clutter if there are alarge number of people within the HMD wearer's field of view. In suchembodiments, eye and gaze tracking or other techniques such as thosedescribed above with reference to the method 200 may be employed todetermine whether the HMD wearer's focal point is on a particularperson. Additionally, as shown at block 304, the length of time theperson is viewed by the HMD wearer is determined. At block 306, it isdetermined whether the length of time satisfies a minimum threshold. Ifso, information is retrieved based on the identified person, as shown atblock 308. If the information is available locally on the HMD, thestored information may be retrieved. Additionally or alternatively, theHMD may communicate with a server to access information. Further, theHMD may communicate with a device (e.g., smartphone, HMD, etc.) of theperson being viewed to obtain information. As discussed previously, theinformation that may be provided for a person may include advertisinginformation and/or personal information regarding that person.Additionally, the information may be retrieved in accordance with accessprivileges set by the person being viewed and/or viewing preferences ofthe HMD wearer. Further factors, such as the context of the interactionbetween the HMD wearer and the person being viewed, may be considered byselecting information.

The information is presented via the HMD, as shown at block 310. Theinformation may be displayed via the HMD, for instance, using HUDtechnology or retinal display technology to allow the HMD wearer to viewthe information. Additionally or alternatively, the information mayinclude audio that is played by speakers on the HMD. Any and all suchcombinations are contemplated to be within the scope of embodiments ofthe present invention.

As discussed previously, in some embodiments, if the HMD wearercontinues to view a particular person, additional information may bepresented to the HMD wearer. Accordingly, the method 300 may continue bydetermining if the HMD wearer continues to view the person, and if so,additional information is retrieved and presented via the HMD.

It should be understood that the method 300 may be performed by an HMDalone or in combination with one or more companion devices (e.g., asmartphone or a dedicated ancillary device) and/or network services(e.g., a cloud-based service). Any portion of the method 300 describedabove may be performed by any combination of these devices and services.

Although the method 300 discussed retrieving information only after theHMD wearer views a person for a threshold period of time, in someembodiments, the HMD may retrieve information in the background for aperson in the HMD wearer's field of view or otherwise known to be in thevicinity of the HMD wearer. This may improve system latency. Inparticular, the information is readily available to be displayed whenthe HMD wearer's focal point remains on a person for the thresholdperiod of time.

HMD Information Based on What Others are Viewing

If multiple people wearing HMDs are looking at the same object orperson, this may indicate that the object or person being viewed isinteresting. Based on this, information could be provided to other HMDwearers in the vicinity, for instance, to bring the object or person tothe other HMD wearers' attention or provide additional informationregarding the object or person.

In embodiments, information may be tracked regarding objects and peopleviewed by HMD wearers. In some embodiments, an object or person may beconsidered to have been viewed by an HMD wearer if the HMD wearer'sfocal point remains on the object or person for a minimum thresholdperiod of time. Additionally, a threshold of HMD wearer's viewing anobject or person may be established that if satisfied for an object orperson, the object or person is considered important. This threshold maybe based on a total number of HMD wearers viewing the object or personor may be based on a percentage of HMD wearers in the area of the objector person who view the object or person. In some embodiments, thethreshold may require the HMD wearers to view the object or person overa common time period. For instance, the HMD wearers may all be lookingat a display in Times Square in New York City at the same time. In otherembodiments, the threshold may allow the HMD wearers to view the objector person at different times. For instance, the system could track whatvisitors to an art museum view over time to identify artwork that isviewed more frequently by visitors.

The identification of objects or people that are viewed by a thresholdof HMD viewers may be employed in a number of different ways in variousembodiments of the present invention. In some embodiments, informationmay be provided to an HMD wearer in the vicinity who is not viewing theobject or person to draw that HMD wearer's attention to the object orperson. This may include animating the object or person to attract theHMD wearer's attention if the object or person is within the HMDwearer's field of view. This may also include providing informationregarding the object or person regardless of whether the object orperson is within the HMD wearer's field of view. The information maysimply indicate that multiple other HMD wearer's are viewing (or viewed)the object or person or may provide more information regarding theobject or person. In further embodiments, an advertisement may beselected based on an object or person viewed by a threshold of HMDwearers and delivered to HMD wearers. This may include delivering theadvertisement to HMD wearers who are viewing the object or person and/orother HMD wearers in the vicinity who are not viewing the object orperson.

In some instances, the object being viewed by a threshold of HMD wearersmay be an advertisement. According to some embodiments, based onidentifying a threshold of HMD wearers viewing an advertisement, theadvertisement may be configured to perform some action. For instance, ifthe advertisement is an electronic billboard, the billboard's displaymay change in response to determining that the billboard is being viewedby a threshold of HMD wearers. In further embodiments, the HMD wearers'view of the advertisement may be augmented by the HMDs in response todetermining that the billboard is being viewed by a threshold of HMDwearers.

Turning to FIG. 4, a flow diagram is provided that illustrates a method400 for providing information to an HMD wearer based on other HMDwearers viewing a particular object or person in accordance with anembodiment of the present invention. Initially, as shown at block 402,an object or person being viewed by HMD wearers is identified. As notedabove, in some embodiments, an object or person may be considered tohave been viewed by an HMD wearer if the HMD wearer's focal pointremains on the object or person for a threshold period of time.

A determination is made at block 404 regarding whether a threshold ofHMD wearers have viewed the object or person. The threshold may be basedon a total number of HMD wearers who have viewed the object or person,or a percentage of HMD wearers who have been in the vicinity of theobject or person and viewed the object or person. The threshold couldalso be based on a comparison of the number of views by HMD wearers ofother object and/or people in the vicinity such that objects and/orpeople viewed more frequently by HMD wearers are identified.Additionally, the threshold may require the HMD wearers to be viewingthe object or person at the same time (e.g., overlapping time period) orsubstantially the same time (e.g., within a few seconds or minutes ofone another) or the threshold may allow the HMD wearers to view theobject or person at different times.

If the threshold is satisfied at block 404 for the object or person,information regarding the object or person is delivered to another HMDwearer based on the determination that the object or person has beenviewed by a threshold of HMD wearers, as shown at block 406. Forinstance, in some embodiments, information may be provided to the HMDwearer to indicate that the object or person is interesting based on HMDwearer views. In further embodiments, advertisements could be selectedbased at least in part on the object or person and delivered to the HMDwearer.

Augmented View of Advertisements

Further embodiments of the present invention are directed to augmentingan HMD wearer's view of an advertisement. The advertisement may be anykind of advertisement displayed in the real world. By way of exampleonly and not limitation, this could include billboards, pressadvertisements (e.g., in magazines, newspapers, etc.), in-storeadvertisements, and electronic advertisements, to name a few. Theadvertisement may have a generic message for people who are not wearingan HMD. However, for users wearing an HMD, the view of the advertisementmay be augmented by the HMD to provide a richer experience.

The view of an advertisement may be augmented in any number of differentmanners within the scope of embodiments of the present invention. Forinstance, in some embodiments, an advertisement may be augmented to drawthe HMD wearer's attention to the advertisement. For example, ananimation may be provided around the advertisement or an image in theadvertisement may become animated to attract the HMD wearer's attention.In such embodiments, the HMD may determine that the advertisement iswithin the HMD wearer's field of view although the HMD wearer's focalpoint is not on the advertisement. As such, the augmentation is added tothe advertisement to draw the HMD wearer's attention. In someembodiments, after the HMD wearer's focal point is placed on theadvertisement, the augmentation may be stopped or a differentaugmentation may be provided (e.g., a personalized view of theadvertisement as discussed below).

In some embodiments, an advertisement may be augmented based onlocation. For instance, suppose that a billboard in Kansas City displaysa general advertisement for NFL football. When the HMD wearer views thebillboard, the view of the advertisement may be augmented by showinginformation and/or other images for the Kansas City Chiefs footballteam.

In further embodiments, the view of an advertisement being viewed by anHMD wearer may be augmented to personalize the advertisement for the HMDwearer. For instance, a user profile of information may be maintainedthat may be employed to provide personalized, augmented views ofadvertisements. The user profile information may include any of avariety of information regarding the HMD wearer, such as demographicinformation, preferences, and interests. For instance, suppose that theHMD wearer is viewing a billboard that has a general message about NFLfootball. Suppose further that the user profile indicates that the HMDwearer is a fan of the Philadelphia Eagles football team. When the HMDwearer views the advertisement, the view of the advertisement may beaugmented such that it displays information and/or other imagesregarding the Eagles team.

An advertisement may also be personalized by augmenting the view of theadvertisement with personalized images. For instance, an advertisementcould be personalized by placing an image of the HMD wearer, the HMDwearer's family, and/or the HMD wearer's friends in the context of theadvertisement. By way of example, suppose that a billboard advertises anamusement park and shows an image of people riding in a roller coaster.When an HMD wearer views the billboard, an augmented view could beprovided in which images of the HMD wearer and members of the HMDwearer's family are shown riding in the roller coaster.

Augmenting an HMD wearer's view of an advertisement may includeaugmenting portions of the advertisement or re-skinning the entireadvertisement in various embodiments of the present invention.Additionally, the augmented view may include, for instance, anycombination of 2-dimensional images, 3-dimensional images, staticimages, or dynamic images.

Turning to FIG. 5, a flow diagram is provided that illustrates a method500 for employing an HMD to augment an HMD wearer's view of anadvertisement to personalize the advertisement in accordance with anembodiment of the present invention. Initially, as shown at block 502,an advertisement being viewed by the HMD wearer is identified. Forinstance, the HMD may include an outward-facing camera that capturesimages/video of the real world around the HMD wearer. Object recognitionmay be employed to identify the advertisement from images/video capturedby the HMD. In some embodiments, the advertisement will only beidentified if the HMD wearer's focal point remains on the advertisementfor a minimum period of time. In other embodiments, any advertisement inthe HMD wearer's field of view or otherwise in the vicinity of the HMDwearer may be identified. For instance, GPS or other locationinformation may be used to determine that a particular advertisement islocated near the HMD wearer.

A user profile for the HMD wearer is accessed at block 504. The userprofile may include a variety of information for the HMD wearer,including, for instance, demographic information, preferences, andinterests. Based on the advertisement identified at block 502 and theuser profile information accessed at block 504, a determination is maderegarding how to augment the HMD wearer's view of the advertisement, asshown at block 506. As discussed previously, any augmentation thatpersonalizes the generic advertisement for the HMD wearer may beemployed. Graphics to augment the advertisement are generated anddisplayed by the HMD, as shown at block 508. Accordingly, the HMD weareris provided a personalized augmented view of the advertisement.

It should be understood that the method 500 may be performed entirely bythe HMD or in conjunction with one or more companion devices (e.g., asmartphone or a dedicated ancillary device) and/or one or more serverdevices (e.g., via a cloud-based service). For instance, the userprofile could be maintained either locally on the HMD, companion device,or may be stored by a server device. Additionally, the determination ofhow to augment the advertisement and/or the generation of graphics couldbe performed by the HMD itself, companion device, or by a server device(which could transmit graphics information to the HMD or otherwiseinstruct the HMD how to augment the view of the advertisement).

Removing or Replacing Advertisements

Given the amount of advertising in today's world, some people may feelinundated with advertisements. To address this issue, in someembodiments, an HMD may augment the view of an advertisement to eitherremove the advertisement or replace the advertisement with othernon-advertising content. As such, while other embodiments discussedabove are directed to augmenting an HMD wearer's view of anadvertisement to enhance the advertising message, the currentlydiscussed embodiments are directed to removing the advertising messagealtogether from the HMD wearer's view.

As noted above, in some embodiments, advertisements may be removed fromthe HMD wearer's view. For instance, the HMD may augment the view of abillboard to make the billboard appear to be blank. However, the HMDviewer would still see the blank billboard. Alternatively, the HMD mayaugment the view of the billboard by making it appear that the billboardisn't there at all. The HMD (or a server device in communication withthe HMD) may determine what computer-generated virtual images should begenerated and displayed to present to the HMD wearer what is behind theadvertisement to make it appear that the advertisement isn't there. Byway of specific example, if the advertisement is a billboard on the sideof a building, the virtual image may represent the side of the buildingto make the building appear without the billboard.

Alternatively, instead of removing an advertisement, the advertisingcontent on the advertisement may be replaced with non-advertisingcontent. In some instances, the non-advertising content may be the HMDwearer's own content. For instance, the content on a billboard could bereplaced with an image from the HMD wearer's photo library. As such,instead of seeing billboards and other advertisements as the HMD wearermoves around the real world, the HMD wearer would see their own imageswhere the advertisements are displayed. In other instances, thenon-advertising content may be content not owned by the HMD wearer. Forinstance, a billboard could be replaced with images from places the HMDwearer has never been. When an advertisement is replaced with othercontent, the content shown may be static content, such as an image, orthe content shown may be dynamic content, such as a video. For example,if the HMD wearer is a football fan, an advertisement could be replacedwith video from a football game. Additionally, the replacement contentmay be 2-dimensional or 3-dimensional content.

The HMD wearer may be provided with the ability to control whichadvertisements are removed or replaced with other content. In someembodiments, the HMD wearer may have preferences specifying whichadvertisements to remove or replace based on the content of theadvertisements. For instance, the user preferences may specify thatadvertisements directed to clothing are to be removed or replaced, whileadvertisements directed to sporting events may be displayed. In someembodiments, the HMD wearer may have preferences for the type ofadvertisement to remove or replace. For instance, the user preferencesmay specify that billboards are removed or replaced, while other typesof advertisements (e.g., press advertisements) are to be displayed.Additionally, the preferences may specify whether the variousadvertisements are removed or replaced with other content. In the caseof replacing advertisements with non-advertising content, thepreferences may specify the non-advertising content to use whenreplacing advertisements. Any and all such combinations of the above arecontemplated to be within the scope of embodiments of the presentinvention.

The ability to remove or replace advertisements with other content maybe a fee-based service provided to users or may be an option provided tousers who are members of some service or group. For instance, theability to remove or replace advertisements may be provided as an optionto users who are Xbox Live members. Additionally, various levels ofcontrol may be given to users based on the fee paid or the membershiplevel of the user (e.g., Xbox Live Silver members or Xbox Live Goldmembers). This may include, for instance, the ability to removeadvertisements, the ability to replace advertisements with othercontent, the ability to control what content is used to replaceadvertisements, and the ability to control content or types ofadvertisements removed or replaced. Generally, a higher fee ormembership level will provide greater control to the user.

Turning to FIG. 6, a flow diagram is provided that illustrates a method600 for employing an HMD to augment an HMD wearer's view of anadvertisement to remove or replace the advertisement in accordance withan embodiment of the present invention. Initially, as shown at block602, an advertisement being viewed by the HMD wearer is identified. Forinstance, the HMD may include an outward-facing camera that capturesimages/video of the world around the HMD wearer. Object recognition maybe employed to identify the advertisement from images/video captured bythe HMD. In some embodiments, the advertisement will only be identifiedif the HMD wearer's focal point remains on the advertisement for aminimum period of time. In other embodiments, any advertisement in theHMD wearer's field of view or otherwise in the vicinity of the HMDwearer may be identified. For instance, GPS or other locationinformation may be used to determine that a particular advertisement islocated near the HMD wearer.

A user profile for the HMD wearer is accessed at block 604. The userprofile may include preferences regarding the removal and/or replacementof advertisements. Based on the advertisement identified at block 602and the user profile information accessed at block 604, a determinationis made regarding whether to remove the advertisement from the HMDwearer's view or to replace the HMD wearer's view of the advertisementwith other content, as shown at block 606. Graphics to remove theadvertisement or replace the advertising content are generated anddisplayed by the HMD, as shown at block 608. Accordingly, theadvertisement is either removed from the HMD wearer's view or theadvertising content is replaced with other content.

The method 600 may be performed entirely by the HMD or in conjunctionwith one or more companion devices (e.g., a smartphone or a dedicatedancillary device) and/or one or more server devices (e.g., via acloud-based service). For instance, the user profile could be maintainedeither locally on the HMD, companion device, or may be stored by aserver device. Additionally, the determination of whether to remove orreplace the advertisement and/or the generation of graphics could beperformed by the HMD itself, companion device, or by a server device(which could transmit the graphics information to the HMD).

Although various embodiments have been described separately herein, itshould be understood that combinations of these may be employed. Forinstance, in some embodiments, an advertisement that is being viewed bymultiple HMD wearers may be brought to the attention of an HMD wearerwho is not directly viewing the advertisement by augmenting the HMDwearer's view to animate the advertisement. Once the HMD wearer viewsthe advertisement (e.g., the HMD wearer's focal point is on theadvertisement), the view of the advertisement may be personalized to theHMD wearer. As another example, the HMD wearer's user profile may causesome advertisements to be removed from the HMD wearer's view while otheradvertisements are augmented to personalize the advertisements to theHMD wearer. Any and all such combinations are contemplated to be withinthe scope of embodiments of the present invention.

As can be understood, embodiments of the present invention provide anumber of approaches for targeting advertising and other information toHMD wearers. The present invention has been described in relation toparticular embodiments, which are intended in all respects to beillustrative rather than restrictive. Alternative embodiments willbecome apparent to those of ordinary skill in the art to which thepresent invention pertains without departing from its scope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects set forth above, togetherwith other advantages which are obvious and inherent to the system andmethod. It will be understood that certain features and subcombinationsare of utility and may be employed without reference to other featuresand subcombinations. This is contemplated by and is within the scope ofthe claims.

What is claimed is:
 1. A head-mounted display (HMD) device comprising: one or more cameras; a display; and a processor configured to perform operations to: identify a real-world advertisement within images captured by the one or more cameras; and generate a computer-generated virtual image and present the computer-generated virtual image via the display to remove the entire real-world advertisement from an augmented view of the real world provided by the display.
 2. The HMD device of claim 1, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world when a focal point of a wearer of the HMD device is on the real-world advertisement.
 3. The HMD device of claim 1, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world based on user preferences to remove real-world advertisements containing particular advertising content and a determination that the real-world advertisement contains the particular advertising content.
 4. The HMD device of claim 1, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world based on user preferences to remove a particular type of real-world advertisement and a determination that the real-world advertisement is of the particular type of real-world advertisement.
 5. One or more tangible computer storage hardware devices storing computer usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising: identifying a real-world advertisement currently within a person's field of view of the real world based on images captured by one or more cameras on an HMD device worn by the person; and in response to identifying the real-world advertisement, causing the HMD device to display a computer-generated virtual image to remove the entire real-world advertisement from an augmented view of the real world provided to the person by the HMD device.
 6. The one or more tangible computer storage hardware devices of claim 5, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device when the HMD device determines the person's focal point is on the real-world advertisement.
 7. The one or more tangible computer storage hardware devices of claim 5, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device based on user preferences to remove real-world advertisements containing particular advertising content and a determination that the real-world advertisement contains the particular advertising content.
 8. The one or more tangible computer storage hardware devices of claim 5, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device based on user preferences to remove a particular type of real-world advertisement and a determination that the real-world advertisement is of the particular type of real-world advertisement.
 9. The one or more tangible computer storage hardware devices of claim 5, wherein the computer generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device based on a fee-based or membership-based service.
 10. The one or more tangible computer storage hardware devices of claim 5, wherein a server device in communication with the HMD device determines how to display the computer-generated virtual image to remove the real-world advertisement from the augmented view of the real world provided by the HMD device.
 11. The one or more tangible computer storage hardware devices of claim 5, wherein a companion device local to the person and in communication with the HMD device determines how to display the computer-generated virtual image to remove the real-world advertisement from the augmented view of the real world provided by the HMD device.
 12. A system comprising: one or more processors; and one or more computer storage media storing computer usable instructions that, when used by the one or more processors, cause the one or more processors to: identify a real-world advertisement currently within a person's field of view of the real world based on images captured by one or more cameras on an HMD device worn by the person; and cause the HMD device to display a computer-generated virtual image to remove the entire real-world advertisement from an augmented view of the real world provided to the person by the HMD device.
 13. The system of claim 12, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device when the HMD device determines the person's focal point is on the real-world advertisement.
 14. The system of claim 12, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device based on user preferences to remove real-world advertisements containing particular advertising content and a determination that the real-world advertisement contains the particular advertising content.
 15. The system of claim 12, wherein the computer-generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device based on user preferences to remove a particular type of real-world advertisement and a determination that the real-world advertisement is of the particular type of real-world advertisement.
 16. The system of claim 12, wherein the computer generated virtual image is displayed to remove the real-world advertisement from the augmented view of the real world provided by the HMD device based on a fee-based or membership-based service.
 17. The system of claim 12, wherein a server device in communication with the HMD device or a companion device local to the person and in communication with the HMD device determines how to display the computer-generated virtual image to remove the real-world advertisement from the augmented view of the real world provided by the HMD device. 